The present invention concerns a method of providing a digitally represented visual instruction from a specialist to a user in need of said visual instruction, and a system therefore. More particularly, the present invention concerns a method of providing a digitally represented visual instruction from a specialist, operating a visual instruction device, to a user in need of said visual instruction in relation to an object of interest and a system therefore. In the method a first video stream comprising images of the object of interest is recorded by a first mobile device associated with the user. The first video stream is sent to a second mobile device associated with the specialist. A second video stream comprising the visual instruction device is recorded by the second mobile device and a foreground/background segmentation operation and a background replacement operation is performed on the second and first video stream to provide a third video stream comprising images of the visual instruction device with the first video stream as background. The third video stream is sent to the first mobile device and is displayed essentially simultaneously on first and second displays on both mobile devices allowing the user, by viewing the first display to be instructed by the specialist, and the specialist, by viewing the second display, to operate the visual instruction device properly in relation to the object of interest.
Methods and systems for providing visual instructions, also known as video collaboration tools or remote specialist assistance systems allow a user such as a service employee, for example a technician or a nurse, to obtain remote specialist assistance by pointing a smartphone towards an object of interest and establishing a video and audio session with a remotely located specialist, so the specialist can see transmitted video of the object of interest and deliver voice instructions to the employee or discuss the object of interest.
One example of a video collaboration or remote assistance tool is disclosed in WO2010108186, according to which a user uses a mobile phone to obtain image data of an object of interest and a remote assistant, using a remote computer, transmits instructions to the mobile phone for assisting the user. The instructions may include a star-shaped cursor, visually pointing to specific features on the object of interest, overlaid upon the image data obtained by the mobile phone and displayed on the mobile phone's display.
However, a cursor is originally designed for a man-machine interface and is an unnatural interface between two people. This makes collaboration less intuitive and more restricted than the ideal situation where the specialist would be physically present standing next to the user so they could both point, gesture and illustrate using their hands, like for example when an experienced medical doctor illustrates palpation manoeuvres to a less experienced colleague.
Further, it is desirable that a remote collaboration session can be established via a mobile device such as a smartphone also at the end of the specialist, so he can be accessed anywhere using familiar technology with no training required and without having to move to a desktop computer or other equipment. Specialists are usually a limited resource and the overhead time involved for them to provide assistance should be minimized.
Other methods and systems for providing remote assistance are disclosed in inter alia U.S. Pat. No. 6,611,242B1 and WO2009128781A1, in which a worker wearing a head mounted camera is instructed by a marker positioned by an instructor, the instructor viewing the image from the camera and the marker being overlaid on the image from the camera and displayed on a head mounted display.
Thus, also the user, and not only the specialist, may be forced in the prior art solutions to use specialized devices.
Some systems perform a keying of the expert's pointing hand into the picture. This eliminates the need for a cursor and a pointing device, like for example an external mouse or a touch screen, the latter having the problem, that it needs to be quite big to avoid that the expert's finger occludes the area of interest when pointing to it. Further, it allows, in addition to simple pointing, instructional gestures to be performed, it is very intuitive and it conveys a feeling of the expert being present.
In many situations, it would be desirable to have a hand keying remote assistance system in the form of an app, i.e. application for a mobile phone or smartphone, which could work on hand held mobile phones without the need for additional accessories.
This would enable instant help from any specialist anywhere, as long as he is carrying his ever-present mobile phone, even if he may be travelling or on-site somewhere else.
Existing hand keying remote assistance systems are designed to work in controlled environments, and require special accessories like a camera mount, a uniformly colored background plate, special gloves or a tracking device etc. Therefore, these systems are not suited for use with hand held mobile phones in the field, where it may unpractical to bring such accessories.
Hence, the unnatural and restricting interface, the overhead time involved and the lack of mobility for the specialists and in some cases the need for specialised devices for the users in the methods and systems of the prior art, are obstacles for wider adoption of remote specialist assistance systems.